Software Methods to Increase Data Cache Performance

Cache performance is critical to the overall performance of modern CPUs. In most processors, cycle time is almost entirely determined by the cache hit time. This places practical limits on the complexity of cache controllers. Because of the growing gap between CPU performance and memory access times, the cost of a cache miss in CPU cycles is growing steadily [1]. As a result, it becomes critical to use caches effectively. More complex replacement schemes have been proposed to predict which data is most likely to be useful in the cache [8]. In addition, complex hardware prefetch schemes have been developed to predict which data is likely to be needed in the future [3,6,7]. However, such modifications are likely to increase the hit time of the cache, and hence the cycle time of the processor. Of particular interest are software schemes to improve cache performance [2,4,5], as they require little or no modifications to the instruction set architecture. In particular, they do not greatly increase the complexity of the cache controller. They also hold the potential to be mined by compilers, having essentially no negative effect on the cost of software development. This paper will examine several such schemes and present their relative costs and benefits. Loop fusion increases the temporal locality of accesses by combining multiple loops that access the same elements. Array merging increases spatial locality by combining multiple arrays whose elements are always accessed together. Cache prefetch reduces cache misses by ensuring that the data is in the cache in advance of when it is needed. The effect of these techniques on example programs is examined.